Audio system and method for coordinating tasks

ABSTRACT

A system includes a hands free mobile communication device. Software stored on a machine readable storage device is executed to cause the hands free mobile communication device to communicate audibly with a field operator performing field operations. The operator receives instructions regarding operations to be performed. Oral communications are received from the operator and are processed automatically to provide further instructions in response to the received oral communications.

BACKGROUND

Handheld devices have been used to support industrial field operationsand maintenance. Such devices have suffered from the use of small userinterfaces, including small format touchscreens and keypads. It isdifficult for users to interact with these devices to enter data via atouch input. The text is small and difficult to read. Bright outdoorlight further obscures the text. Further, while performing operationsthat require the use of both hands, the device must be continuallystowed and retrieved. Since some operations are performed on ladders orplatforms and sometimes in inclement weather, the interaction with thedevice may be very frustrating and distract the user from performingfield operations.

SUMMARY

A system includes one or more hands free mobile communication devices.Software stored on a machine readable storage device is executed tocause the hands free mobile communication devices to communicate audiblywith field operators performing field operations. The operators areinstructed regarding operations to be performed. Oral communications arereceived from the operators, interpreted by the system and routedappropriately. The interpreted communications may automatically triggerfurther instructions.

Oral communications may include comments from the operator. The systemin one embodiment may route or otherwise process the comments as afunction of the context in which the comments are generated and thecontent of the comment.

In some embodiments, the system supports the synchronous execution ofprocedures by multiple operators and by multiple operators incoordination with an automated control system.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a portrayal of a field operator in an industrial plantperforming a task in a procedure utilizing a hands free communicationdevice according to an example embodiment.

FIG. 2 is a block diagram of a controller interacting with multiplehands free mobile communication devices and a console operator interfaceaccording to an example embodiment.

FIG. 3 is a flowchart illustrating a method of handling commentsaccording to an example embodiment.

FIG. 4 is a flowchart illustrating handling of audible instructions froma field operator according to an example embodiment.

FIG. 5 is a drawing of a field operator interface for a hands freemobile communication device according to an example embodiment.

FIG. 6 is a drawing a portion of a field operator interface for a handsfree mobile communication device according to an example embodiment.

FIG. 7 is a drawing of a portion of a field operator interface for ahands free mobile communication device according to an exampleembodiment.

FIGS. 8A, 8B, and 8C taken together illustrate a chart describinginteraction of a console operator and a field operator interacting witha procedure according to an example embodiment.

FIG. 9 is a block diagram of a computer system that executes programmingfor executing code to control procedures performed by operatorsaccording to an example embodiment.

DETAILED DESCRIPTION

In the following description, reference is made to the accompanyingdrawings that form a part hereof, and in which is shown by way ofillustration specific embodiments which may be practiced. Theseembodiments are described in sufficient detail to enable those skilledin the art to practice the invention, and it is to be understood thatother embodiments may be utilized and that structural, logical andelectrical changes may be made without departing from the scope of thepresent invention. The following description of example embodiments is,therefore, not to be taken in a limited sense, and the scope of thepresent invention is defined by the appended claims.

The functions or algorithms described herein may be implemented insoftware or a combination of software and human implemented proceduresin one embodiment. The software may consist of computer executableinstructions stored on computer readable media such as memory or othertype of storage devices. Further, such functions correspond to modules,which are software on a storage device, hardware, firmware or anycombination thereof. Multiple functions may be performed in one or moremodules as desired, and the embodiments described are merely examples.The software may be executed on a digital signal processor, ASIC,microprocessor, or other type of processor operating on a computersystem, such as a personal computer, server or other computer system.

FIG. 1 is a portrayal 100 of a field operator 110 in an industrial plant115, such as a refinery, performing a task in a procedure for handlingoperations and maintenance of the industrial plant. In one embodiment,the field operator 110 is on a ladder, and is using a hands free mobilecommunication device 120, such as a cellular phone or integrated dataand voice communications device along with a wireless headset 125 tointeract with a system to jointly execute the procedure. A procedure maybe thought of as a set of explicit actions or steps and instructionsthat are followed by operational personnel alone or in closecoordination with an automated control system and optionally multiplepersonnel in a coordinated manner to execute a process or achieve agoal. While an industrial plant is used as an example, the invention mayalso be adapted to operations and maintenance of space vehicles,maintenance and servicing of commercial aircraft on the ground, and insecurity and HVAC servicing and operations in buildings.

The procedure may be written in a machine executable program that isstored on a storage device, and provides speech based instructions tothe operator. The system generates the speech based instructions, andreceives oral data and commands from the field operator 110 as shown ina dialog box 130. The speech generated instructions from the system areprovided to the operator. The first instruction in this example dialog130 is “Confirm Tag number.” The operator responds orally with “11AC12”.The system interprets the oral response and acts in accordance with theprocedure, such as perhaps checking the tag number for accuracy and thenprovides a question for the operator to answer: “Is discharge pressuregreater than 90 psi?” The operator responds with “No”. This response isalso interpreted and may affect the flow of the procedure. Given theresponse of “No”, the next instruction is “Check delta P for filters.”The operator responds “Delta P is 50%”. An instruction may then beprovided to “Change filters”. The field operator in this example wasable to stay on the ladder, and complete a procedure without having tophysically interact with the device.

Many different procedures may be facilitated by the system, such asstructured tasks, maintenance rounds, procedural operations such as unitstart ups and shut downs, product installations and servicing. Thesystem is fully speech interactive and wireless using a body worn mobiledevice.

In one embodiment, as shown in block diagram 200 in FIG. 2, theinstructions describing one or more procedures may be stored andexecuted on a controller 210, which may be referred to as a procedurecontroller. The controller 210 may be a stand alone device, or coupledto or part of a process controller, such as a distributed processcontroller 212 that controls operations in a plant. Further, theinstructions may be stored on one or more handheld devices that allowhands free communication with a field operator.

Controller 210 in one embodiment communicates with multiple operatorsvia devices 215, 220 and 225. Each device may be a wireless mobilecommunications platform with a wireless or wired headset having aspeaker and microphone device indicated at 230, 235 and 240 respectivelyto provide a hands free mobile communication device. In furtherembodiments, noise cancellation features may be included, as manyindustrial plants like refineries can have a high level of ambient noisethat would otherwise interfere with communications. In still furtherembodiments, the mobile communications platform may be integrated withthe headset.

In further embodiments, controller 210 may have a console operatorinterface 250 coupled to it via a wired or wireless network. Consoleoperator interface 250 may be a common workstation in one embodiment,having a display, keyboard, and a data entry device (e.g., mouse,touchpad, etc.), and is also coupled to the distributed control system212. Still further, operator interface 250 may include voicecapabilities for communicating directly with operators and the controldevice in a manner similar to that provided by the mobile communicationdevices 215, 220, and 225. Operator interface 250 may also be used aspart of the execution of a procedure program being executed bycontroller 210. For instance, the operator interface 250 may be used toallow an operator to interact directly with a control system to causeactions to occur in the plant that are relevant to tasks being performedby one or more field operators. In one simple example, a valve may needto be closed using the controller 210 prior to the field operatorperforming a measurement or other action.

In one embodiment, controller 210 may execute a procedure that includesinstructions for causing the hands free mobile communication device tocommunicate audibly with a field operator performing field operations.The mobile communication device may instruct the operator regardingoperations to be performed, receive oral communications from theoperator, and to interpret/process the received oral communications bymeans of speech recognition and provide further instructions responsiveto the received oral communications.

In some procedures, multiple field operators may be helping to performthe procedure. The activities of the operators may need to besynchronized. One operator may need to complete a task prior to anotheroperator starting a succeeding task. Some tasks may be performed inparallel by two or more operators. Operators may need to perform tasksin synchrony with tasks performed by the automatic process controlsystem. In this case, an operator may have to wait for the controlsystem to perform an action, such as ramping up temperature or pressurein a vessel to a prescribed level, before he or she opens a valve,introduces another chemical to the process, etc. A procedure authoringtool 260, such as one called PRIDE, may be used to generate proceduresin a high level language, such as an XML script, for execution. Theprocedure authoring tool allows the programming of if-then scenarios, asillustrated above, where the field operator was instructed to replace afilter when a difference in pressure read in the field exceeded acertain threshold. The XML script, or other type of machine executablelanguage, may also control timing of tasks, recognition of responses,and allow for comments.

In one embodiment, the script identifies text to read to a fieldoperator, either by text to speech conversion, or by a pre-recordedaudio file (e.g., way, etc.). The script then causes a machine executingit to perform speech recognition on audible information and commandsreceived from the operator. Oral communications are recognized andconverted to text commands. The format of the script in one exampleembodiment may take the form of {Instr.;Response;Comment}. The followingis a word list corresponding to commands for hands free interaction forone example form that the procedure language may take in one exampleembodiment:

# word list file, put each voice command on a separate line # anythingafter a ′#′ symbol is ignored by parser Ok Cancel Phonetic Numbers#Unlimited Digits 0-9 Save and Exit On Off Phonetic Alphabets #UnlimitedLetters inc “space” Clear Next Previous Edit Clear Text Check Box SkipCheck Previous Settings User Settings Status Page Reset Values All AutoManual All Data Message Comment One Done Invalid Pause Comment Resume

In one embodiment, one of the text commands causes recording of audiblecomments provided by the field operator as illustrated in a flowchart300 in FIG. 3. The text commands for handling comments may includeComment, done, invalid, pause, and resume as indicated above or othercommands necessary to control the recording of comments or notes. Suchcommands may be operable to initiate a comment at 310 and while audiblecomments are being recorded. At 320, the audible comment is converted totext, and at 330, the comment text is analyzed to identify or inferautomatically a recipient of an electronic communication to which thecomment will be sent. In further embodiments, the comment may simply berecorded in a comment log.

Keywords in the comment may be identified in the text and compared toknown keywords to identify the recipient and to create the communicationbased on the comment as indicated at 340. The communication in oneembodiment may include a context in which the comment is entered. In oneexample, the word “broken” or “needs repair” may result in a work orderbeing prepared as part of the communication. From the context of thetask, such as “inspect pipe”, the work order may be routed to a personor department that is responsible for fixing pipes. There are many otherexamples that are easy to generate given a list of potential repairs, orother actions that would be known to result from a specific procedure.In some embodiment, the context includes a name of a procedure beingperformed by the field operator at the time the comment is provided.

FIG. 4 is a flowchart illustrating a method 400 that includes providingaudible instructions to a field operator via a hands free communicationdevice in accordance with a procedure written in a machine readable format 410. Oral commands may be received from the hands free mobilecommunication device at 420 as an operator performs the procedure inaccordance with the audible instructions. At 430, the oral commands areconverted to machine readable form to continue executing the procedureinteractively via the hands free communication device.

In one embodiment, the procedure may synchronize instructions andcommands via multiple hands free communication devices such thatmultiple operators are instructed to perform tasks in the procedure in asynchronized manner. Furthermore, multiple operators may be instructedto perform tasks in synchrony with tasks performed by the automatedprocess control system. Some tasks may be performed in parallel andother tasks are performed only after receiving oral commands indicatingthat a predetermined task is completed. The tasks may be assigned tospecific field operators.

In further embodiments, a handheld mobile communication device mayinclude a graphical user interface, one instance of which is illustratedat 500 in FIG. 5. The graphical user interface 500 may include itemssuch as step title 505, previous step 510, previous instruction 515,current step 520, current instruction 525, response 530 and status, suchas pause, comment, etc., at 535. Further menus may be accessed via anApp Control button 540 and an Options button 545. Such menus areillustrated at 610 in FIG. 6, and 710 in FIG. 7 respectively.

The user interface 500 may be used to help a field operator keep trackof where they are in a procedure in case the operator becomes distractedduring performance of the procedure. It may also be used to enter datavia a touchscreen interface or other interface other than audio ifdesired. The graphical user interface, touchscreen interface, or otherinterface works in synchrony with the speech-based interface so that theuser can move seamlessly back and forth between interaction modalities.In some embodiments, speech recognition may be performed at the mobilecommunication device or the controller 210.

FIGS. 8A, 8B, and 8C together form a chart 800 illustrating theinteraction of a console operator with a field operator, facilitated bythe use of a hands free mobile communication device using the system andmethod of performing procedures described above. In this example, aprocedure has been initiated to handle a coker heater tube rupture inCoker C 1234 as indicated in a notes column 810. A console operatorcolumn 820 provides instructions on a screen, in audio, or both to aconsole operator, usually residing in a control room. Completion for theconsole operator may be by button or audio if desired. Operatorinstructions and interactions are detailed in column 830, with anobserved result indicated in column 840.

The console operator is first instructed to contact the field operatorto inform him or her about the procedure that will be executed, and asksthe field operator to confirm arrival at the Coker. The field operatorconfirms arrival and may be provided with an indication to wait for thenext instruction. Waiting music may be provided, or further informationregarding the procedure to be performed may be provided in variousembodiment. The console operator then performs several operationsresponsive to instructions provided by execution of the procedure. Stepsin chart 800 are numbered in the left most column. Once parameters reacha predetermined level as indicated in step 5 a., a beep or othercommunication may be provided to the field operator to get ready toperform a task. The first task for the field operator is identified inan audible instruction to verify that a furnace has tripped to naturaldraft. The operator may reply with a confirmatory command, such as “OK”,or “Verified”.

As can be seen in this procedure, operations by the console operator andfield operator transfer back and forth. In step 9, another operator maybe asked to perform a task, followed by further instructions for thefield operator in column 830. In a mixed manual-automated procedure, theautomated control system may perform tasks as well, exchanginginitiative as dictated by the procedure with the field and consoleoperators.

A block diagram of a computer system that executes programming forperforming the procedures is shown in FIG. 9. A general computing devicein the form of a computer 910, may be specifically programmed, and mayinclude a processing unit 902, memory 904, removable storage 912, andnon-removable storage 914. Memory 904 may include volatile memory 906and non-volatile memory 908. Computer 910 may include—or have access toa computing environment that includes—a variety of computer-readablemedia, such as volatile memory 906 and non-volatile memory 908,removable storage 912 and non-removable storage 914. Computer storageincludes random access memory (RAM), read only memory (ROM), erasableprogrammable read-only memory (EPROM) & electrically erasableprogrammable read-only memory (EEPROM), flash memory or other memorytechnologies, compact disc read-only memory (CD ROM), Digital VersatileDisks (DVD) or other optical disk storage, magnetic cassettes, magnetictape, magnetic disk storage or other magnetic storage devices, or anyother medium capable of storing computer-readable instructions. Computer910 may include or have access to a computing environment that includesinput 916, output 918, and a communication connection 920. The computermay operate in a networked environment using a communication connectionto connect to one or more remote computers. The remote computer mayinclude a personal computer (PC), server, router, network PC, a peerdevice or other common network node, or the like. The communicationconnection may include a Local Area Network (LAN), a Wide Area Network(WAN) or other networks.

Computer-readable instructions to execute methods and algorithmsdescribed above may be stored on a computer-readable medium such asillustrated at a program storage device 925 are executable by theprocessing unit 902 of the computer 910. A hard drive, CD-ROM, and RAMare some examples of articles including a computer-readable medium.

The Abstract is provided to comply with 37 C.F.R. §1.72(b) is submittedwith the understanding that it will not be used to interpret or limitthe scope or meaning of the claims.

1. A system comprising: a hands free mobile communication device; acontroller programmed to cause the hands free mobile communicationdevice to communicate audibly with a field operator performing fieldoperations to instruct the operator regarding operations to beperformed, receive oral communications from the operator, and to processthe received oral communications and provide further instructionsresponsive to the received oral communications.
 2. The system of claim 1wherein the controller further automatically recognizes oral commandsfrom the operator and provides further instructions based on the stateof procedure being executed by the automated process control system. 3.The system of claim 1 and further comprising a storage device to store aprogram for execution by the controller.
 4. The system of claim 1 andfurther comprising multiple hands free mobile communication devicescoupled to the controller, wherein the controller coordinates multiplefield operators via the devices with each other and with a state of anautomated control system.
 5. The system of claim 1 wherein the oralcommunications are recognized and converted to text commands and whereinthe communication device is operable to receive and transmit non-oralinput from the operator and switch seamlessly between oral and non-oralinput modalities.
 6. The system of claim 5 wherein one of the textcommands causes recording of audible comments provided by the fieldoperator.
 7. The system of claim 6 wherein the text commands furtherinclude done, pause, and resume commands operable while audible commentsare being recorded.
 8. The system of claim 6 wherein the content andprocedural context of a comment is analyzed to identify a recipient ofan electronic communication including the comment.
 9. The system ofclaim 8 wherein keywords in the comment are used to identify therecipient, and the communication includes a context in which the commentis entered.
 10. The system of claim 9 wherein the context includes aname of a procedure being performed by the field operator at the timethe comment is provided.
 11. The system of claim 6 wherein a comment isanalyzed to generate a work order as a function of the content of thecomment.
 12. The system of claim 1 wherein the audible instructionsprovided to instruct the operator are synthesized speech generated fromthe instructions.
 13. A method comprising: providing audibleinstructions to a field operator via a hands free communication devicein accordance with a procedure written in a machine readable form;receiving oral commands from the hands free mobile communication deviceas an operator performs the procedure in accordance with the audibleinstructions; and converting the oral commands to machine readable formto continue executing the procedure interactively via the hands freecommunication device.
 14. The method of claim 13 and further comprisingsynchronizing instructions and commands via multiple hands freecommunication devices and a state of an automated control system suchthat multiple operators are instructed to perform tasks in the procedurein a manner that is synchronized with the automated control system andwith each other operator.
 15. The method of claim 14 wherein some tasksare performed in parallel and other tasks are performed only afterreceiving oral commands indicating that a predetermined task iscompleted.
 16. The method of claim 15 wherein tasks are assigned tospecific field operators and to the automated control system.
 17. Amachine readable device having instructions stored thereon to cause amachine to perform a method comprising: providing audible instructionsto a field operator via a hands free communication device in accordancewith a procedure embodied in the instructions; receiving oral commandsfrom the hands free mobile communication device; and converting the oralcommands to machine readable form to coordinate performance of theprocedure interactively via the hands free communication device.
 18. Themachine readable device of claim 17 wherein the method further comprisessynchronizing instructions and commands via multiple hands freecommunication devices such that multiple operators are instructed toperform tasks in the procedure in a synchronized manner and in a mannersynchronized with the automated control system.
 19. The machine readabledevice of claim 17 wherein some tasks are performed in parallel andother tasks are performed only after receiving oral commands indicatingthat a predetermined task is completed.
 20. The machine readable deviceof claim 17 wherein tasks are assigned to specific field operators.